Skip to content
This repository was archived by the owner on Aug 2, 2021. It is now read-only.

cmd/swarm: new localstore database migration#1297

Merged
nonsense merged 18 commits intolocalstore-storage-integrationfrom
ls-integration-migration
Apr 10, 2019
Merged

cmd/swarm: new localstore database migration#1297
nonsense merged 18 commits intolocalstore-storage-integrationfrom
ls-integration-migration

Conversation

@acud
Copy link
Copy Markdown
Contributor

@acud acud commented Mar 13, 2019

This PR handles and tests the flow of migrating from the old localstore to the new one.
The update sequence is as follows:

  1. User updates the Swarm binary once this gets released
  2. Swarm binary checks for old schema name and identifies when the old chunk store format is present and quits the process with exit code 1. User gets alerted. The binary won't start before datastore is migrated
  3. User should create a DB export using swarm db export
  4. User should delete/move old chunk store (/chunks)
  5. User starts the Swarm binary, new empty chunkstore gets created, user should quit the Swarm binary at this point
  6. User should run the DB import command with --legacy flag
  7. Datastore migration should be now complete

At a minimum the disk space toll is @ x2 and x3 at most. We have with @janos decided we do not want to automate this process and we also don't want to add an automatic migration flag to the binary because this will require users to change the way they run Swarm and this change might not be reverted in some cases and can cause trouble in the future (i.e. a User adds a --migrate flag to her runswarm.sh script and forgets to remove it, cause future migrations to potentially be executed automatically).

RTFM: https://hackmd.io/UAV2uybkQlSerVIX5xUF0g?view

@acud acud force-pushed the ls-integration-migration branch from f805f78 to 793e3e6 Compare March 14, 2019 11:28
@janos janos mentioned this pull request Mar 15, 2019
@acud acud marked this pull request as ready for review March 19, 2019 09:07
@acud acud force-pushed the ls-integration-migration branch 3 times, most recently from 896d485 to a4df16e Compare March 19, 2019 09:34
@zelig zelig mentioned this pull request Mar 20, 2019
26 tasks
Comment thread swarm/storage/localstore/schema.go Outdated
Comment thread swarm/swarm.go Outdated
Comment thread cmd/swarm/testdata/datastore_fixture.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be shorter?

Comment thread cmd/swarm/flags.go Outdated
acud added 15 commits March 28, 2019 20:41
cmd/swarm: add ldb fixture for integration testing, add gzip deflate

cmd/swarm: pull deflate to helper function

cmd/swarm: start swarm with old datastore

cmd/swarm: test assert that process exits with non-zero exit code on old db

cmd/swarm: assert export size

cmd/swarm: remove datadir

cmd/swarm: change export verbosity

cmd/swarm: fix TestSomething

cmd/swarm: rename migration test

cmd/swarm: wip e2e test

cmd/swarm: wip e2e test

swarm: legacy schema detection
@acud acud force-pushed the ls-integration-migration branch from 4f8cd74 to 8b61fde Compare March 28, 2019 11:58
…t, remove unnecessary error log line that might confuse users, simplify legacy db check
@acud
Copy link
Copy Markdown
Contributor Author

acud commented Mar 28, 2019

@nonsense I have the results for a dry run migration on my local machine. I'm running an i5 CPU and my SSD is a SATA III SSD, so performance is not so high.
Datastore size was at around 5.5gb.
I added a resource pooling to throttle the db.Puts. I tried 100 and 200 for pool size without a big difference, so I'm pushing 100 at the moment.
tl;dr: export takes 1 minute for 5 gigs, import takes about 15m

export
time ./swarm_mig db export /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/exphttps://github.com/ethersphere/go-ethereum/pull/1297/commits/33e1bb277f193c34e68cb93091762ab59385f85dort.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321                                                                                    
INFO [03-28|21:45:56.713] successfully exported 1375147 chunks 
./swarm_mig db export  /home/growlster/migrationTest/export.tar   24.01s user 19.84s system 78% cpu 56.153 total

import (token pool @ 100)
time ./swarm_mig db import /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/export.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321
INFO [03-28|22:08:13.592] successfully imported 1375147 chunks 
./swarm_mig db import  /home/growlster/migrationTest/export.tar   1134.67s user 131.82s system 164% cpu 12:50.79 total

import (token pool @ 200)
time ./swarm_mig db import /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/export.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321
INFO [03-28|22:29:11.723] successfully imported 1375147 chunks 
./swarm_mig db import  /home/growlster/migrationTest/export.tar   1134.91s user 132.98s system 165% cpu 12:47.45 total

@nonsense nonsense merged commit 904df7a into localstore-storage-integration Apr 10, 2019
@acud acud deleted the ls-integration-migration branch April 11, 2019 03:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants